Camera system and traveling control system

ABSTRACT

A camera system including an imaging device that captures a first image by a normal exposure including only one exposure and that captures a second image by a multiple exposure including a plurality of exposures; and a control circuit. The the imaging device captures the second image in a first frame period, and the control circuit determines, based on the second image captured in the first frame period, whether to capture an image by the normal exposure or capture an image by the multiple exposure in a second frame period following the first frame period.

CROSS-REFERENCE OF RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.16/257,587, filed on Jan. 25, 2019, which in turn claims the benefit ofJapanese Application No. 2018-021977, filed on Feb. 9, 2018, the entiredisclosures of which Applications are incorporated by reference herein.

BACKGROUND 1. Technical Field

The present disclosure relates to a camera system.

2. Description of the Related Art

A camera system is known which includes an imaging device configured toacquire a multiple exposure image by performing an exposure a pluralityof times. For example, Japanese Unexamined Patent ApplicationPublication No. 2002-27315 discloses a camera system in which a featurevalue of a subject is extracted from a multiple exposure image acquiredby an imaging device, and a location of the subject in the multipleexposure image is identified based on the extracted feature value.

SUMMARY

There is a need for a camera system capable of more accuratelyidentifying a location of a subject in a multiple exposure image.

In one general aspect, the techniques disclosed here feature a camerasystem including an imaging device that captures a first image by anormal exposure including only one exposure and that captures a secondimage by a multiple exposure including a plurality of exposures; and acontrol circuit, where the imaging device captures the second image in afirst frame period, and the control circuit determines, based on thesecond image captured in the first frame period, whether to capture animage by the normal exposure or capture an image by the multipleexposure in a second frame period following the first frame period.

It should be noted that general or specific embodiments may beimplemented as an element, a device, an apparatus, a camera system, anintegrated circuit, a method, a computer program, or a computer-readablestorage medium in which a program is stored. It should be noted thatgeneral or specific embodiments may be implemented by any selectivecombination of an element, a device, an apparatus, a camera system, anintegrated circuit, a method, a computer program, or a computer-readablestorage medium in which a program is stored.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration of a travelingcontrol system according to a first embodiment;

FIG. 2 is a schematic diagram illustrating an example of a circuitconfiguration of an imaging device according to the first embodiment;

FIG. 3 is a circuit diagram illustrating an example of a circuitconfiguration of one pixel according to the first embodiment;

FIG. 4 is a cross-sectional view schematically illustrating a structureincluding a photoelectric conversion element and an FD according to thefirst embodiment;

FIG. 5 is a circuit diagram illustrating an example of a circuitconfiguration of one pixel according to the first embodiment;

FIG. 6 is a diagram illustrating an example of operation timing in anormal exposure and a multiple exposure in two frame periods in animaging device according to the first embodiment;

FIG. 7A is a diagram illustrating an example of a multiple exposureimage acquired by an imaging device according to the first embodiment;

FIG. 7B is a diagram illustrating an example of a reference imageacquired by an imaging device according to the first embodiment;

FIG. 8 is a diagram illustrating an example of operation timing in anormal exposure and a multiple exposure in two frame periods in animaging device according to the first embodiment;

FIG. 9 is a flow chart illustrating a first moving object detectionprocess;

FIG. 10 is a conceptual diagram schematically illustrating a manner ofextracting a feature value from a subject;

FIG. 11 is a diagram illustrating an example of a multiple exposureimage;

FIG. 12 is a diagram illustrating an example of a multiple exposureimage;

FIG. 13 is a diagram illustrating operation timing in nondestructivereading of a charge by an imaging device according to the firstembodiment;

FIG. 14 illustrates an example of a multiple exposure image;

FIG. 15 is a flow chart illustrating a second moving object detectionprocess;

FIG. 16 is a flow chart illustrating a third moving object detectionprocess;

FIG. 17 is a flow chart illustrating a fourth moving object detectionprocess;

FIG. 18 is a diagram illustrating an example of operation timing in amultiple exposure in two frame periods in an imaging device according toa fourth embodiment;

FIG. 19A is a diagram illustrating an example of operation timing in anormal exposure and a multiple exposure in two frame periods in animaging device according to a fifth embodiment;

FIG. 19B is a diagram illustrating an example of operation timing in anormal exposure and a multiple exposure in two frame periods in animaging device according to the fifth embodiment;

FIG. 19C is a diagram illustrating an example of operation timing in anormal exposure and a multiple exposure in two frame periods in animaging device according to the fifth embodiment;

FIG. 20A is a diagram illustrating an example of operation timing in anormal exposure and a multiple exposure in two frame periods in animaging device according to a sixth embodiment;

FIG. 20B is a diagram illustrating an example of operation timing of anormal exposure and a multiple exposure in two frame periods in animaging device according to the sixth embodiment;

FIG. 21 is a diagram illustrating an example of operation timing in anormal exposure and a multiple exposure in two frame periods in animaging device according to a seventh embodiment;

FIG. 22A is a diagram illustrating an example of a multiple exposureimage acquired via a first moving object detection process;

FIG. 22B is a diagram illustrating an example of a multiple exposureimage acquired via a seventh moving object detection process; and

FIG. 23 is a block diagram illustrating a configuration of a travelingcontrol system according to a modified embodiment.

DETAILED DESCRIPTION Underlying Knowledge Forming Basis of the PresentDisclosure

In a case where a subject moves relative to an imaging device, when amultiple exposure image is acquired by the imaging device, a pluralityof subject images are superimposed at locations shifted from each other.In such a situation, a conventional camera system may make an error indetecting a shape of or the number of subject images from a multipleexposure image. Such an erroneous detection may result in a reduction inaccuracy in identifying a location of a subject image in a multipleexposure image. In view of the above, the present disclosure provides acamera system in various aspects as summarized below.

Aspect 1

A camera system includes an imaging device that acquires a first imageby a normal exposure including only one exposure and that acquires asecond image by a multiple exposure including a plurality of exposures;and

an image processor that extracts a feature value of a first object inthe first image and that identifies one or more locations correspondingto the feature value in the second image.

Aspect 2

In the camera system described in Aspect 1, the imaging device mayacquire the first image in a first frame period and that acquires thesecond image in a second frame period following the first frame period.

Aspect 3

The camera system described in Aspect 2 may further include a controlcircuit that determines, based on the second image, whether to acquirean image by the normal exposure or acquire an image by the multipleexposure in a third frame period following the second frame period.

Aspect 4

In the camera system described in Aspect 4, in a case where the one ormore locations corresponding to the feature value cannot be identifiedin the second image, the control circuit determines to acquire an imageby the normal exposure in the third frame period.

Aspect 5

In the camera system described in Aspect 3 or 4, in a case where asecond object different from the first object is detected in the secondimage, the control circuit determines to acquire an image by the normalexposure in the third frame period.

Aspect 6

In the camera system described in Aspect 3 or 4, in a case where asecond object different from the first object is detected in a firstarea of the second image, the control circuit determines to acquire animage by the normal exposure in the third frame period.

Aspect 7

In the camera system described in one of Aspects 1 to 6, the imagingdevice may nondestructively read out a signal generated by an exposureout of the multiple exposure.

Aspect 8

In the camera system described in Aspect 7, the imaging device mayinclude pixels and nondestructively reads out the signal from a pixelout of the pixels.

Aspect 9

In the camera system described in Aspect 1,

the imaging device may acquire the first image and the second image in afirst frame period, and

the imaging device may acquire the first image by a first exposure outof the multiple exposure and acquires the second image by the multipleexposure. the imaging device.

Aspect 10

In the camera system described Aspect 9, the imaging device maynondestructively read out a signal generated in the first exposure.

Aspect 11

In the camera system described in one of Aspects 1 to 10,

the multiple exposure may include a first exposure and a second exposuredifferent from the first exposure, and

a sensitivity in the first exposure may be different from a sensitivityin the second exposure.

Aspect 12

In the camera system described in one of Aspects 1 to 11, a sensitivityin a period between adjacent two exposures out of the multiple exposuremay be greater than zero and less than a sensitivity in each of theadjacent two exposures.

Aspect 13

In the camera system described in one of Aspects 1 to 12, the firstobject may include at least a car, a motorcycle, a train, a bicycle, aperson, or an animal.

Aspect 14

In the camera system described in one of Aspects 1 to 13, the imageprocessor may calculate, based on the second image, at least a directionof movement, a velocity, or an acceleration of the first object.

Aspect 15

The camera system described in Aspect 14 may further include a displayapparatus that displays at least the direction of movement, thevelocity, or the acceleration calculated by the image processor.

Aspect 16

In the camera system described in Aspect 1, the image processor maydetect, based on the feature value of the first object, one or moreobject images corresponding to the first object in the second image.

Aspect 17

A traveling control system includes

the camera system according to claim 1; and

a controller, wherein

the camera system calculates, based on the second image, at leastdirection of movement, a velocity, or an acceleration of the firstobject, and

the controller controls, based on the at least the direction ofmovement, the velocity, or the acceleration calculated by the camerasystem, a travel state of a moving body.

Aspect 18

A camera system includes;

an imaging device that acquires a first image by one or more exposuresand that acquires a second image by a multiple exposure including aplurality of exposures; and

an image processor that extracts a feature value of a first object inthe first image and that identifies one or more locations correspondingto the feature value in the second image.

Aspect 19

In the camera system described in Aspect 18;

the imaging device may acquire the first image and the second image in afirst frame period, and

the imaging device may acquire the first image by first one or moreexposures out of the multiple exposure and acquires the second image bythe multiple exposure.

Aspect 20

In the camera system described in Aspect 18, the image processor maydetect, based on the feature value of the first object, one or moreobject images corresponding to the first object in the second image.

In an aspect of the present disclosure, a camera system includes animaging device that acquires an image and an image processor thatperforms an image processing, wherein the imaging device acquires afirst image by performing one exposure and acquires a second image byperforming a plurality of exposures, and the image processor extracts afeature value of a first object from the first image and identifies alocation of the first object in the second image based on the featurevalue.

This camera system extracts a feature value of the first object, thatis, the subject, from the first image which is not a multiple exposureimage. Using the extracted feature value, a plurality of images of thefirst object are extracted from the multiple exposure image. Thus, thiscamera system is capable of reducing the probability of erroneouslydetecting the shape of and/or the number of the subject images comparedwith the conventional camera system configured to extract a featurevalue of a subject image from a multiple exposure image.

Thus, this camera system is capable of providing an improved accuracy inidentifying a location of a subject in a multiple exposure image.

For example, the imaging device may perform the first-time exposure inthe first frame period and may perform the plurality of exposures in thesecond frame period following the first frame period.

This makes it possible to start extracting the feature value of thefirst object before the acquisition of the second image is started or inparallel to the acquisition of the second image.

For example, the camera system may further include a control circuitthat controls the imaging device. The control circuit may determine,based on the second image, whether the imaging device is to perform theone exposure or the plurality of exposures in a third frame periodfollowing the second frame period.

This makes it possible to dynamically determine whether the exposureperformed in the third frame period is to be one exposure or a pluralityof exposures.

For example, in a case where the image processor detects no first objectin the second image, the control circuit may determine that the imagingdevice is to perform the one exposure in the third frame period.

This makes it possible to acquire, in the third frame period, an imagefor extracting a feature value of a new object.

For example, in a case where the image processor detects a second objectdifferent from the first object in the second image, the control circuitmay determine that the imaging device is to perform the one exposure inthe third frame period.

This makes it possible to acquire, in the third frame period, an imagefor extracting the feature value of the second object.

For example, in a case where the image processor detects a second objectdifferent from the first object in a predetermined particular area whichis part of the second image, the control circuit may determine that theimaging device is to perform the one exposure in the third frame period.

This makes it possible to reduce the amount of processing in the processof determining whether to perform one exposure or a plurality ofexposures in the third frame period.

For example, nondestructive reading may be performed on a chargeobtained by at least one exposure in the plurality of exposure.

This makes it possible to acquire an image obtained by the at least oneexposure and an exposure previous to the at least one exposure.

For example, the nondestructive charge reading may be performed on partof pixels included in the imaging device.

This makes it possible to reduce the reading time in the nondestructivereading and/or the power consumption in the nondestructive readingcompared with a case where the nondestructive reading is performed onall pixels included in the imaging device.

For example, the one exposure may be a first exposure in the pluralityof exposures, and the imaging device may perform the plurality ofexposures in the same frame period.

This makes it possible to start extracting the feature value of thefirst object before the acquisition of the second image is started or inparallel to the acquisition of the second image.

For example, the charge obtained by the one exposure may be read outnondestructively.

This makes it possible to superimpose an image of a subject acquired bythe one exposure on the second image.

For example, the plurality of exposures may include a first exposureperformed with a first exposure sensitivity and a second exposureperformed with a second exposure sensitivity different from the firstexposure sensitivity.

This makes it possible to obtain images of a subject superimposed on thesecond image such that the luminance of the image of the object acquiredby the first exposure is different from the luminance of the image ofthe subject acquired by the second exposure.

For example, the plurality of exposures each may be a high-sensitivityexposure performed with a particular exposure sensitivity, and theimaging device may perform the plurality of exposures such that thehigh-sensitivity exposure and the low-sensitivity exposure are performedalternately and successively.

This makes it possible to form, on the second image, a trajectoryindicating time series locations of a subject.

For example, the first object may be one of a car, a motorcycle, atrain, a bicycle, a person, and an animal.

This makes it possible to identify a location of a car, a motorcycle, atrain, a bicycle, a person, or an animal on the second image.

For example, the image processor may calculate at least one of adirection of movement, a velocity, and an acceleration of the firstobject based on the second image.

This makes it possible to calculates at least one of the direction ofmovement, the velocity, and the acceleration of the first object.

For example, the camera system may further include a display apparatusthat displays at least one of the direction of movement, the velocity,and the acceleration.

This makes it possible for a user of the camera system to visuallyrecognize at least one of the direction of movement, the velocity, andthe acceleration of the first object.

According to an aspect, the present disclosure provide a vehicletraveling control system that controls traveling of a vehicle. Thetraveling control system includes a camera system, wherein the imageprocessor calculates at least one of a direction of movement, avelocity, and an acceleration of the first object based on the secondimage, and the vehicle traveling control system performs the controldescribed above based on at least of one the direction of movement, thevelocity, and the acceleration.

Thus, the vehicle traveling control system is capable of controlling thetraveling of the vehicle based on at least one of the direction ofmovement, the velocity, and an acceleration of the first object.

Specific examples of the camera system and the traveling control systemaccording to the aspect of the present disclosure are described belowwith reference to drawings. Note that each embodiment described below isfor illustrating a specific example of the present disclosure.Therefore, in the following embodiments of the present disclosure,values, shapes, constituent elements, locations of elements, manners ofconnecting elements, steps, the order of steps, and the like aredescribed by way of example but not limitation.

Among constituent elements described in the following embodiments, thoseconstituent elements that are not described in independent claims areoptional. Note that each drawing is a schematic diagram, which does notnecessarily provide a strict description.

First Embodiment

A configuration of a camera system according to a first embodiment and aconfiguration of a traveling control system according to the firstembodiment are described below with reference to drawings.

1.1 Configuration

FIG. 1 is a block diagram illustrating a configuration of a travelingcontrol system 1. The traveling control system 1 is installed in avehicle and controls traveling of the vehicle.

As shown in FIG. 1, the traveling control system 1 includes a camerasystem 10, and an electronic control unit 80.

The camera system 10 includes an imaging device 20, an image processor30, a control circuit 40, an optical system 50, an image transmissionunit 60, and a display apparatus 70.

The optical system 50 includes a set of lens including a plurality oflenses. The optical system 50 focuses light incident from the outside ofthe camera system 10 onto the imaging device 20. The set of lenses mayinclude a focus lens. The focus lens may adjust a focal position of asubject image in the imaging device 20 by moving the set of lenses in anoptical axis direction.

The imaging device 20 acquires an image. The imaging device 20 may be,for example, a CMOS (Complementary Metal Oxide Semiconductor) imagesensor.

FIG. 2 is a schematic diagram illustrating an example of a circuitconfiguration of the imaging device 20. In the example shown in FIG. 2,it is assumed that the imaging device 20 is a CMOS image sensor.

As shown in FIG. 2, the imaging device 20 includes a plurality of pixels101 arranged in the form of a matrix, a row scanning circuit 102, acolumn scanning circuit 103, current sources 104 provided in respectivecolumns, and AD (Analog to Digital) conversion circuits 105 provided inrespective columns.

Pixels 101 located in each one of the rows are connected together to therow scanning circuit 102 via a common horizontal signal line 107. Pixels101 located in each one of the columns are connected together to thecolumn scanning circuit 103 via a common vertical signal line 108 and acommon AD conversion circuit 105. Power is supplied to each pixel 101via a power supply line 106 shared by all pixels 101.

In the example of the configuration shown in FIG. 2, the AD conversioncircuits 105 are provided in the respective columns such that a digitalsignal is output from the column scanning circuit 103. However, the ADconversion circuits 105 may not be provided, and an analog signal may beoutput from the column scanning circuit 103. The column scanning circuit103 may include, in each column, an adder/subtractor circuit thatperforms addition or subtraction on the signals given from pixels 101located in the column and outputs a result of the addition orsubtraction operation.

Each pixel 101 includes a photoelectric conversion element that performsa photoelectric conversion. The photoelectric conversion element may be,for example, of a type using a photoelectric conversion film or a typerealized using a photodiode formed in a semiconductor substrate.

FIG. 3 is a circuit diagram illustrating an example of a circuitconfiguration of one pixel 101. In FIG. 3, the pixel 101 includes aphotoelectric conversion element using a photoelectric conversion film.

As shown in FIG. 3, the pixel 101 includes a photoelectric conversionelement 21, a floating diffusion layer (hereinafter, referred to as“FD”) 22, an amplifier transistor 23, a selection transistor 24, and areset transistor 25.

The photoelectric conversion element 21 performs a photoelectricconversion on incident light. The FD 22 accumulates a charge generatedin the photoelectric conversion element 21.

FIG. 4 is a cross-sectional view schematically illustrating a structureincluding the photoelectric conversion element 21 and the FD 22 shown inFIG. 3.

As shown in FIG. 4, the photoelectric conversion element 21 includes athin-film photoelectric conversion layer 110C, a thin-film transparentelectrode 110A located above the photoelectric conversion layer 110C,and a thin-film pixel electrode 1106 located below the photoelectricconversion layer 110C. The pixel electrode 1106 is connected, via acontact plug 110E, to the FD 22 formed in the semiconductor substrate110D. The FD 22 is, for example, a diffusion layer including animpurity.

When light is incident on the photoelectric conversion layer 110C in astate in which a bias voltage is applied between the transparentelectrode 110A and the pixel electrode 110B, charges are generated by aphotoelectric effect. Of the generated charges, a charge of eitherpositive or negative type is collected by the pixel electrode 110B. Thecharge collected by the pixel electrode 110B is accumulated in the FD22.

By changing the bias voltage applied to the photoelectric conversionlayer 110C, it is possible to change a photoelectric conversionefficiency of the photoelectric conversion layer 110C, that is, thesensitivity of the photoelectric conversion layer 110C. Morespecifically, for example, when the bias voltage applied to thephotoelectric conversion layer 110C is relatively high, thephotoelectric conversion efficiency, that is, the sensitivity, of thephotoelectric conversion layer 110C is relatively high. Conversely, whenthe bias voltage applied to the photoelectric conversion layer 110C isrelatively low, the photoelectric conversion efficiency, that is, thesensitivity, of the photoelectric conversion layer 110C is relativelylow.

Furthermore, by changing the bias voltage applied to the photoelectricconversion layer 110C such that the bias voltage is smaller than aparticular threshold value, it is possible to set the photoelectricconversion efficiency of the photoelectric conversion layer 110C to besubstantially equal to 0. Hereinafter, this threshold value is referredto as an exposure threshold value.

That is, by changing the bias voltage applied to the photoelectricconversion layer 110C between a value greater than the exposurethreshold value and a value smaller than the exposure threshold value,it is possible to change the state of the photoelectric conversionelement 21 between an exposure state and a non-exposure state. Thismakes it possible for the imaging device 20 to perform a multipleexposure operation in which an exposure is performed a plurality oftimes intermittently.

Referring again to FIG. 3, the pixel 101 is further explained.

When the reset transistor 25 is turned on-state by applying a Vrstsignal, the electric potential of the FD 22 is reset to a desired resetpotential V1.

The amplifier transistor 23 outputs a signal corresponding to the amountof charge accumulated in the FD 22.

By switching the state of the selection transistor 24 between theon-state and the off-state using a Vsel signal, a selection is made asto whether a signal output from the amplifier transistor 23 is output tothe vertical signal line 108 or not.

FIG. 5 is a circuit diagram illustrating another example of a circuitconfiguration of one pixel 101. In the example shown in FIG. 5, thepixel 101 includes a photodiode functioning as the photoelectricconversion element.

As shown in FIG. 5, the pixel 101 includes a photodiode 26, a floatingdiffusion layer (hereinafter referred to as “FD”) 22, an amplifiertransistor 23, a selection transistor 24, a reset transistor 25, atransfer transistor 27, and a discharge transistor 28.

The photodiode 26 performs a photoelectric conversion on incident light.

When the transfer transistor 27 is switched into the on-state inresponse to the Vtrs signal, the charged generated in the photodiode 26is transferred to the FD 22.

When the discharge transistor 28 is switched into the on-state inresponse to the Vdr signal, the charge generated in the photodiode 26 isdischarged.

In the imaging device 20, of a total charge generated in the photodiode26, a charge arising from an exposure intended to be effective istransferred to the FD 22 via the transfer transistor 27 while a chargearising from an exposure intended to be ineffective is discharged viathe discharge transistor 28. This makes it possible to perform amultiple exposure operation in which an exposure is performed aplurality of times intermittently.

Referring again to FIG. 1, the traveling control system 1 is furtherexplained.

The image processor 30 performs various processes on the image acquiredby the imaging device 20. A first moving object detection processperformed by the image processor 30 will be described later withreference to a flow chart.

As shown in FIG. 1, the image processor 30 includes an image signalprocessor (hereafter referred to as “ISP”) 31, and a motion detectionunit 32.

The ISP 31 mainly performs a correction process and/or the like.

The motion detection unit 32 mainly performs an extraction of a featurevalue and processes associated with a moving object, such as a motiondetection, a moving object detection, and/or the like on an image outputfrom the ISP 31. The motion detection unit 32 includes, for example, aprocessor which executes a program to achieve the processes associatedwith the moving object. The motion detection unit 32 may include a framememory for storing an image and/or a memory for storing a program, anextracted feature value and/or other data.

The image transmission unit 60 outputs data such as image data inputfrom the ISP 31 to an external unit. The external unit is, for example,the electronic control unit 80. The image data output from the imagetransmission unit 60 may be, for example, uncompressed and unprocessedraw data or data in a particular format subjected to image compressionand/or particular image processing.

The control circuit 40 controls the camera system 10. The controlcircuit 40 also controls the imaging device 20.

As shown in FIG. 1, the control circuit 40 may include, for example, anoutput unit 41, an input unit 42, a microcomputer 45, a program memory43 for storing a program to be executed by the microcomputer 45, and aworking memory 44 used by the microcomputer 45.

The display apparatus 70 displays information obtained as a result ofthe process performed by the motion detection unit 32. The displayapparatus 70 may display, for example, at least one of the direction ofmovement, the velocity, and the acceleration calculated by the motiondetection unit 32.

The electronic control unit 80 is a unit called an ECU (Engine ControlUnit). The electronic control unit 80 controls an engine, braking,accelerating, and/or the like of a vehicle in which the electroniccontrol unit 80 is installed. The electronic control unit 80 may performthe control described above based on, for example, at least one of thedirection of movement, the velocity, and the acceleration calculated bythe motion detection unit 32.

An operation performed by the camera system 10 configured in theabove-described manner is described below with reference to drawings.

1.2 Operation 1.2.1 Image Capturing

The operation of capturing an image by the imaging device 20 isdescribed in further detail below. The description is given below withreference to drawings for each of cases: a case in which thephotoelectric conversion element is of a type using a photoelectricconversion film; and a case in which a photodiode is used as thephotoelectric conversion element.

FIG. 6 illustrates an example of operation timing in a normal exposureand a multiple exposure in two frame periods. Note that each pixel 101of the imaging device 20 includes a photoelectric conversion elementusing a photoelectric conversion film.

In FIG. 6, VD represents a change in vertical synchronization signal VDhaving a pulse form and changing at a predetermined frame rate (forexample, 60 fps).

In synchronization with timing of rising of VD, the imaging device 20reads out a pixel signal from each pixel 101 thereby acquiring an image.The pixel signal corresponds to the amount of charge accumulated in theFD 22.

In FIG. 6, VITO represents a change in the bias voltage VITO applied tothe transparent electrode 110A. The bias voltage VITO can also beregarded as a voltage applied to the photoelectric conversion layer110C. The bias voltage VITO is set alternately to a low level lower thanthe exposure threshold value and a high level higher than the exposurethreshold value.

In the example shown in FIG. 6, VITO is set to the high level once in afirst frame period while VITO is set to the high level three times in asecond frame period following the first frame period. Thus, each pixel101 has one exposure period in the first frame period and three exposureperiods in the second frame period. That is, an image is acquired as aresult of one exposure in the first frame period, while an image isacquired as a result of a plurality of exposures in the second frameperiod. Hereinafter, one exposure referred to a normal exposure and aplurality of exposures performed in one frame period is referred to as amultiple exposure. An image acquired by a normal exposure is referred asa reference image, and an image acquired by a multiple exposure isreferred to as a multiple exposure image. For example, when an image isacquired by reading an amount of charge accumulated in the FD 22 in thefirst frame period in which an exposure occurs once in each pixel 101, areference image is obtained as a result. When an image is acquired byreading an amount of charge accumulated in the FD 22 in the second frameperiod in which an exposure occurs a plurality of times in each pixel101, a multiple exposure image is obtained as a result. For example, ina case where a subject moves relative to the imaging device 20, amultiple exposure image includes a plurality of subject imagessuperimposed at locations shifted from each other in a directioncorresponding to a direction in which the subject moves. On the otherhand, a reference image include only single subject image regardless ofwhether a subject moves relative to the imaging device 20. In otherwords, a reference image cannot be an image including a plurality ofsubject images superimposed at locations shifted from each other in adirection corresponding to a direction in which the subject moves.

FIG. 7A illustrates an example of a multiple exposure image acquired byan imaging device 20 installed on a vehicle. FIG. 7B illustrates anexample of a reference image acquired by the imaging device 20 installedon the vehicle.

FIG. 7A and FIG. 7B each are an example of an image ahead of thevehicle, acquired in a state in which another vehicle 201 and threemotorcycles 211, 212, and 213 having similar external appearancestraveling ahead of the vehicle. The vehicle 201 moves at a particularrelative speed with respect to the vehicle, while the relative speed ofeach of motorcycles 211, 212, and 213 with respect to the vehicle issubstantially zero.

As shown by way of example in FIG. 7A, in the multiple exposure image, aplurality of images of the same vehicle 201 are formed such that theimages are formed at locations shifted from each other in a relativemovement direction of the vehicle 201. On the other hand, a single imageis obtained for each of the three motorcycles 211, 212, and 213 even inthe multiple exposure image. Therefore, it may be relatively difficultfrom this multiple exposure image to determine whether the images ofthree motorcycles 211, 212, and 213 are images of different threemotorcycles or images of one same motorcycle obtained by multipleexposure.

In contrast, in the reference image, as shown by way of example in FIG.7B, a plurality of subject images are never obtained for one samesubject.

FIG. 8 illustrates an example of operation timing of the imaging device20 in a normal exposure and a multiple exposure in two frame periods. Inthis example, it is assumed that each pixel 101 of the imaging device 20includes a photodiode functioning as the photoelectric conversionelement.

In FIG. 8, VD represents a change in vertical synchronization signal VDhaving a pulse form and changing at a predetermined frame rate (forexample, 60 fps).

In synchronization with timing of rising of VD, the imaging device 20reads out a pixel signal from each pixel 101 thereby acquiring an image.The pixel signal corresponds to the amount of charge accumulated in theFD 22.

In FIG. 8, Vtrs represents a change in a pulse signal Vtrs that controlsthe transfer transistor 27. When Vtrs is at the high level, the transfertransistor 27 is in the on-state, while when Vtrs is at the low level,the transfer transistor 27 is in the off-state. In other words, in aperiod in which Vtrs is at the high level, a charge generated in thephotodiode 26 is transferred to the FD 22, while in a period in whichVtrs is at the low level, a charge generated in the photodiode 26 is nottransferred to the FD 22.

In FIG. 8, Vdr represents a change in a pulse signal Vdr that controlsthe discharge transistor 28. When Vdr is at the high level, thedischarge transistor 28 is in the on-state, while when Vdr is at the lowlevel, the discharge transistor 28 is in the off-state. In other words,in a period from the beginning of a frame period to a time immediatelybefore Vdr rises to the high level and a period from a time immediatelyafter Vtrs falls to the low level to a time immediately before Vdr risesto the high level, a charge generated in the photodiode 26 isdischarged, while in a period in which Vdr is at the low level, a chargegenerated in the photodiode 26 is not discharged.

Thus, an exposure period is given by a period from the falling-down ofVdr to the next falling-down of Vtrs.

In the example shown in FIG. 8, Vdr and Vtrs are set to the high levelonce in a first frame period while Vdr and Vtrs are set to the highlevel three times in a second frame period. In both periods, Vdr firstrises to the high level and then Vtrs rises to the high level. Thus,each pixel 101 has one exposure period in the first frame period andthree exposure periods in the second frame period. That is, an image isacquired as a result of one exposure in the first frame period, while inthe second frame period, an image is acquired as a result of a pluralityof exposures. When an image is acquired by reading an amount of chargeaccumulated in the FD 22 in the first frame period in which an exposureoccurs once in each pixel 101, a reference image is obtained as aresult. When an image is acquired by reading an amount of chargeaccumulated in the FD 22 in the second frame period in which an exposureoccurs a plurality of times in each pixel 101, a multiple exposure imageis obtained as a result.

1.2.2 First Moving Object Detection Process

The camera system 10 performs a first moving object detection process.

The first moving object detection process is described below withreference to drawings.

FIG. 9 is a flow chart illustrating the first moving object detectionprocess.

The first moving object detection process starts, for example, when thecamera system 10 accepts an operation performed by a user of the camerasystem 10 to set the operation mode of the camera system 10 to a movingobject detection mode.

As shown in FIG. 9, when the first moving object detection processstarts, the imaging device 20 acquires a reference image by performing anormal exposure (step S100). An example of a reference image obtainedhere is shown in FIG. 7B.

When the reference image is acquired, the image processor 30 extracts afeature value of each of all subjects from the reference image. Based oneach extracted feature value, the image processor 30 determines whethereach subject is of a specific type whose motion vector is to beextracted (step S110). Hereinafter, a subject of the specific type whosemotion vector is to be extracted is referred to as a “specific subject”.

The feature value is a quantitative feature of a shape and is used inchecking or identifying a subject. Examples of feature values include anedge pattern, a luminance pattern, a color pattern, and an aspect ratio.The edge pattern includes, for example, locations of vertices of thesubject, the number of vertices, a gradient direction of an edge, and anangle of the edge. The luminance pattern is a quantified featurerepresenting a distribution of luminance of an image. The color patternis a quantified feature representing a distribution of color of animage. The aspect ratio represents a ratio of a width to a height of animage, or the like, quantified as a feature value. Other examples offeature values focused on relationships between local areas are a SIFT(Scale Invariant Feature Transform) feature value, a HOG (Histogram oforiented gradient) feature value, an EOH (Edge of OrientationHistograms) feature value, an Edgelet feature value, etc. Still othervarious feature values may also be extracted.

The specific subject is, for example, an object that can autonomouslymove. Specific examples of specific subjects are a car, a motorcycle, atrain, a bicycle, a person, and an animal.

In the traveling control system 1, the motion vector is extracted foronly a moving object having a power source and a living body capable ofautonomously moving. This makes it possible to effectively monitor anobject that can be a danger in the traveling of the vehicle, whilereducing the amount of processing. Examples of moving objects having apower source are cars, motorcycles, trains, and bicycles. Examples ofliving bodies capable of autonomously moving are persons, and animals.FIG. 10 schematically illustrates a manner of extracting feature values1000 a to 1000 n of subjects from a reference image.

The image processor 30 recognizes what the subjects are, from theextracted feature values (step S120).

The image processor 30 may have a table in which a definition of arelationship between a specific subject and its feature value is storedin advance, and the image processor 30 may recognize the subject basedon the degree of similarity between the extracted feature value andfeature values stored in the table. Hereinafter, a subject recognized inthe above-described manner is referred to as a “first object”. The imageprocessor 30 stores the extracted feature value of the first object.

When a subject is recognized, the imaging device 20 acquires a multipleexposure image by performing a multiple exposure (step S130).

In the above description, it is assumed by way of example that theprocess in step S130 is performed after the processes from step S110 tostep S120 are completed. However, the imaging device 20 and the imageprocessor 30 can operate independently, and thus the processes from stepS110 to step S120 can be executed in parallel to the process in stepS130.

When the multiple exposure image is acquired, the image processor 30extracts a first object in the multiple exposure image based on thestored feature value (step S140). More specifically, for example, theimage processor 30 performs template matching using the stored featurevalue as a template thereby extracting the first object. The imageprocessor 30 may identify a location, in the multiple exposure image,corresponding to the stored feature value.

When the first object is extracted from the multiple exposure image, theimage processor 30 identifies the location of the first object in themultiple exposure image (step S150).

The image processor 30 then calculates a direction of movement, avelocity, and an acceleration of the first object from the identifiedlocation of the first object in the multiple exposure image (step S160).

A method of calculating the direction of movement, velocity, and theacceleration is described below.

Calculating Direction of Movement

A specific example of a method of calculating the direction of movementof the first object from the multiple exposure image is described below.

Locations of images of the first object corresponding to time points atintervals are identified from the multiple exposure image. Thus, if atime difference between each adjacent locations can be known, it ispossible to calculate the direction of movement of the first object. Ina short time period, the first object gradually moves away from thestarting point of the movement. Therefore, when the location of thestarting point is known, it can be estimated that the closer to thestarting point, the older the location is, which makes it possible tocalculate the direction of movement. In a case where a one-frameprevious image is given as a reference image, the location of the firstobject in this reference image may be employed as a start point of themovement, and the direction of movement of the first object can becalculated from the multiple exposure image. In a case where a one-frameprevious image is not a reference image but a multiple exposure image,an ending point of the first object in this one-frame previous multipleexposure image may be employed as a start point of the movement, and thedirection of movement of the first object can be calculated from thecurrent multiple exposure image.

Calculating Velocity

A specific example of a method of calculating the velocity of the firstobject from the multiple exposure image is described below.

FIG. 11 illustrates an example of a multiple exposure image acquired byperforming exposure a plurality of times (three times in the specificexample) at intervals of time t. In this multiple exposure image, thefirst object is a car.

As shown in FIG. 11, a displacement d of the location of the firstobject in the multiple exposure image is calculated. If the size of thefirst object in the actual space can be known, it is possible todetermine the actual distance D of the movement of the first object inthe actual space for a time 1. The time t corresponds to the exposurecycle t shown in FIGS. 6 and 8. The size of the first object in theactual space can be estimated with reference to a known size such as anumber plate of a vehicle or the like whose size is specified by astandard. The size of the first object in the actual space may beroughly estimated from the type of the first object (such as a car,person, etc.). If the actual distance D can be known, then it ispossible to calculate the velocity of the first object by dividing theactual distance D by time t.

In the above-described example of the method of calculating the velocityof the first object, the velocity is calculated based on the location ofthe first object captured in the first exposure in the plurality ofexposures and the location of the first object captured in the secondexposure in the plurality of exposures. However, it is possible tocalculate the velocity of the first object based on locations of thefirst object captured by arbitrary two exposures. Furthermore, thevelocity may be calculated based on locations of the first objectobtained by combinations each including two arbitrary two exposures in aplurality of exposures, and an average value thereof may be employed asthe velocity of the first object.

Note that the value of the velocity obtained via the calculationdescribed above is a relative velocity of the first object withreference to the imaging device 20 in a case where the imaging device 20is moving. In a case where the imaging device 20 is at rest, thecalculated value indicates the absolute velocity of the first object.

Calculating Acceleration

A specific example of a method of calculating the acceleration of thesecond object from the multiple exposure image is described below.

As described above, a velocity may be calculated based on locations ofthe first object obtained by an arbitrary set of two exposures in aplurality of exposures in a multiple exposure image. This method may beused to calculate velocity of the first object for two differentperiods. Furthermore, the acceleration of the first object may becalculated from the difference in velocity between the two differenceperiods.

Referring again to FIG. 9, the first moving object detection process isfurther described below.

When the process in step S160 is completed, the camera system 10determines whether to end the moving object detection mode (step S170).

For example, in a case where the camera system 10 accepts an operationperformed by a user of the camera system 10 to set the operation mode ofthe camera system 10 to a mode other than the moving object detectionmode, the camera system 10 may determine that the moving objectdetection mode is to be ended. On the other hand, for example, in a casewhere the camera system 10 receives, from the electronic control unit80, a signal indicating that the engine has stopped, the camera system10 may determine that the moving object detection mode is to be ended.

In a case where it is determined in the process in step S170 that themoving object detection mode is not to be ended (No in step S170), thecontrol circuit 40 determines whether to perform a normal exposure or amultiple exposure in a next frame period (step S180).

Next, a method of determining whether to perform a normal exposure or amultiple exposure in a next frame period is determined below.

Determining Exposure Mode in Next Frame Period

Once a feature value of a first object is extracted from a referenceimage, this feature value can be used repeatedly in identifyinglocations of the first object in a following multiple exposure image.

However, in a case where the first object disappears from the followingmultiple exposure image or in a case where a new object (hereinafterreferred to as a “second object”) different from the first objectappears, it is necessary to newly extract a feature value from thereference image.

Therefore, the control circuit 40 determines, based on the multipleexposure image, whether the imaging device 20 is to perform a normalexposure or a multiple exposure in a next frame period. Morespecifically, for example, in a case where the first object is notdetected in the multiple exposure image, or in a case where a secondobject is detected in the multiple exposure image, the control circuit40 determines that the normal exposure is to be performed in the nextframe period. However, in any other cases, the control circuit 40determines that the multiple exposure is to be performed in the nextframe period.

The first object may disappear from the multiple exposure image, forexample, in one of the following cases: a case in which the first objectmoves to the outside of the imaging area of the imaging device 20; acase in which a change in view of the first object occurs due to achange in the direction or the angle of the first object; a case inwhich another object overlaps the first object and thus the first objectis hidden behind the object located closer to the imaging device 20 thanthe location of the object; etc.

An occurrence of disappearance of the first object may be detected, forexample, by checking whether the number of images of the first objectdetected on the multiple exposure image is smaller than the number ofexposures performed in the image capturing operation in the multipleexposure mode.

FIG. 12 illustrates an example of a multiple exposure image.

As can be seen from FIG. 12, a second object may enter the imaging areaof the imaging device 20 through an upper edge area 200 a, a lower edgearea 200 b, a left edge area 200 c, or a right edge area 200 d. Thus, inpixel regions corresponding to the areas described above, nondestructivereading may be performed in each period between adjacent exposures in amultiple exposure. This makes it possible to detect entering of a secondobject into the imaging area of the imaging device 20. Hereinafter, thearea for detecting the entering of the second object will also bereferred to as a “moving object detection area”.

FIG. 13 illustrates operation timing in nondestructively reading of anamount of charge accumulated in FD 22 from a pixel 101 in a movingobject detection area. Note that it is assumed here that each pixel 101of the imaging device 20 includes a photoelectric conversion elementusing a photoelectric conversion film.

In FIG. 13, a graph of a nondestructive reading signal illustratestimings of pulse signals that cause it to start nondestructively readingout an amount of charge accumulated in the FD 22 from pixels 101. Insynchronization with timing of rising of the nondestructive readingsignal from the low level to the high level, the charge accumulated inthe FD 22 is nondestructively read out.

As described above, by performing nondestructive reading between eachadjacent exposures in a multiple exposure, it is possible to detect achange in location of an object entering imaging area of the imagingdevice 20, and thus it is possible to easily detect the object in amultiple exposure image. Although there is no particular restriction onthe size of the moving object detection area, it is desirable todetermine the size of the moving object detection area such that a goodbalance is achieved between the amount of processing and the size of theobject.

In the example described above with reference to the drawings, it isassumed by way of example that the nondestructive reading is performedin each period between adjacent exposures in a multiple exposure.However, the nondestructive reading may be performed only once in aperiod between particular adjacent exposures.

In a case where the imaging device 20 is installed on a vehicle, theupper region above the imaging area of the imaging device 20 is the sky.In such a case, the upper edge area 200 e in the moving object detectionarea may be set to be located closer to the center of the imaging areaas shown in FIG. 14.

The moving object detection area is not limited to a fixed area, but themoving object detection area may be variable.

Referring again to FIG. 9, the first moving object detection process isfurther described below.

In a case where it is determined in the process in step S180 that anormal exposure is to be performed in a next frame period, the camerasystem 10 returns to the process in step S100 (Yes in step S190). Inthis case, the process is repeated from step S100.

In a case where it is not determined in the process in step S180 that anormal exposure is to be performed in a next frame period, that is, in acase where it is determined that a multiple exposure is to be performed,the camera system 10 returns to the process in step 130 (No in stepS190). In this case, the process is repeated from step S130.

In a case where it is determined in the process in step S170 that themoving object detection mode is to be ended (Yes in step S170), thecamera system 10 ends the first moving object detection process.

1.3 Supplementary Remarks

The camera system 10 is capable of detecting a relative direction ofmovement of a first object by tracking a change in the location of thefirst object in a multiple exposure image. In a case where the actualsize of the first object, the focal length of the optical system 50, andthe exposure cycle in the multiple exposure are known, it is possible tocalculate the relative velocity, the relative acceleration, and/or thelike of the first object.

As described above, the camera system 10 extracts a feature value of thefirst object from a first image which is not a multiple exposure image.Using the extracted feature value, the first object is extracted fromthe multiple exposure image. Thus, it is possible to reduce aprobability that the camera system 10 erroneously detects the shape, thenumber, and/or the like of the first object.

Thus, it becomes possible for the camera system 10 to accurately andprecisely identify the location of the first object in the multipleexposure image.

Second Embodiment 2.1 Overview

A camera system according to a second embodiment is described below.

In the first embodiment described above, a multiple exposure image isacquired after a reference image is acquired.

In the second embodiment, unlike the first embodiment, after a referenceimage is acquired, it is determined, based on the acquired referenceimage, whether a reference image or a multiple exposure image isacquired next.

2.2 Configuration

In the second embodiment, the camera system may be similar inconfiguration to the camera system 10 according to the first embodiment,and thus a description of the constituent elements is omitted.

2.3 Operation

In the camera system according to the second embodiment, a second movingobject detection process is performed instead of the first moving objectdetection process according to the first embodiment.

The second moving object detection process is described below withreference to drawings.

FIG. 15 is a flow chart illustrating the second moving object detectionprocess.

As shown in FIG. 15, the second moving object detection process isdifferent from the first moving object detection process according tothe first embodiment in that the process in step S120 is replaced by aprocess in step S220.

The following explanation focuses on the process in step S220 andrelated processes.

When a feature value of a specific subject is extracted from a referenceimage in the process in step S110, the image processor 30 tries torecognize what the specific subject on the reference image is, based onthe extracted feature value of the specific subject (step S220).

In a case where the specific subject is recognized in the process instep S220 (Yes in step S220), the processing flow proceeds to theprocess in step S130 to acquire a multiple exposure image.

In a case where the specific subject cannot be recognized in the processin step S220 (No in step S220), the processing flow proceeds to theprocess in step S100 to acquire a reference image.

2.4 Supplementary Remarks

In the camera system according to the second embodiment, the referenceimage acquisition process is repeated until a specific subject isrecognized from a reference image in the process in step S220, that is,until an object that can be a danger in the traveling of the vehicle isrecognized. A multiple exposure image is acquired only when a specificsubject is recognized from a reference image in the process in stepS220.

In the camera system according to the second embodiment, as describedabove, in a case where an object that can be a danger in traveling ofthe vehicle is not detected from a reference image, the multipleexposure image acquisition is not performed.

Thus, the camera system according to the second embodiment is capable ofeffectively monitoring an object that can be a danger in traveling of avehicle while reducing the amount of processing for detecting a movingobject.

Third Embodiment 3.1 Overview

A camera system according to a third embodiment is described below.

In the first embodiment described above, each time a multiple exposureimage is acquired, a determination is made as to whether to acquire areference image or a multiple exposure image in a next frame period, anda next image is acquired according to a determination result.

In contrast, in the third embodiment, after a reference image isacquired once, a multiple exposure image is acquired successively apredetermined number of times. Furthermore, the acquisition of thereference image and the acquisition of the multiple exposure image areperformed alternately.

3.2 Configuration

In the third embodiment, the camera system may be similar inconfiguration to the camera system 10 according to the first embodiment,and thus a description of the constituent elements is omitted.

3.3 Operation

In the camera system according to the third embodiment, a third movingobject detection process is performed instead of the first moving objectdetection process according to the first embodiment.

The third moving object detection process is described below withreference to drawings.

FIG. 16 is a flow chart illustrating the third moving object detectionprocess.

As shown in FIG. 16, the third moving object detection process isdifferent from the first moving object detection process according tothe first embodiment in that a process in step S300 is added before theprocess in step S100, the process in step S180 is replaced by a processin step S380, and the process in step S190 is replaced by a process instep S390.

The following explanation is given while focusing on the process in stepS300, the process in step S380, the process in step S390, and relatedprocesses.

As shown in FIG. 16, when the third moving object detection processstarts, the control circuit 40 initializes an integer type variable n(step S300). Herein, the initialization of the variable n is achieved bysubstituting 0 into the variable n.

After the variable n is initialized, the processing flow proceeds to aprocess in step S100 to acquire a reference image.

In a case where it is determined in the process in step S170 that themoving object detection mode is not to be ended (No in step S170), thecontrol circuit 40 substitutes n+1 into the variable n (step S380).

The control circuit 40 checks whether the new substituted value of thevariable n is equal to a preset integer X greater than or equal to 1(step S390). That is, the control circuit 40 the number of times themultiple exposure image has been successively acquired has reached thepreset value X.

The value X may be set, for example, depending on the traveling speed ofthe vehicle on which the imaging device 20 is installed. The value X maybe set depending on a type of a road on which the vehicle, on which theimaging device 20 is installed, travels. The value X may be setdepending on weather around the vehicle on which the imaging device 20is installed. Examples of types of roads are a highway, an ordinal road,etc. Examples of weather are fine weather, rainy weather, etc.

In a case where it is determined in the process in step S390 that a newvalue substituted into the variable n is equal to the integer X (Yes instep S390), the processing flow proceeds to step S100 to repeat theprocess from step S300. That is, the variable n is initialized (stepS300), and a reference image is captured (step S100). Note that when thenew value substituted into the variable n is equal to the integer X, thenumber of times the multiple exposure image has been successivelyacquired reaches the preset particular value X.

In a case where it is determined in the process in step S390 that thenew value substituted into the variable n is not equal to the integer X(No in step S390), the processing flow proceeds to step S130 to repeatthe process from step S130. That is, the successive acquisition of themultiple exposure image is further continued. Note that when the newvalue substituted into the variable n is not equal to the integer X, thenumber of times the multiple exposure image has been successivelyacquired has not yet reached the preset particular value X.

3.4 Supplementary Remarks

In the first embodiment described above, each time a multiple exposureimage is acquired, a determination is made as to whether to perform anormal exposure or a multiple exposure in a next frame period.

In contrast, in the third embodiment, the determination as to whether toperform a normal exposure or a multiple exposure in a next frame periodis not made each time a multiple exposure image is acquired.

Thus, in the camera system according to the third embodiment, it ispossible to further reduce the amount of processing in detecting amoving object to a level smaller than is achieved in the camera systemaccording to the first embodiment.

As described above, in the camera system according to the thirdembodiment, after a reference image is acquired once, a multipleexposure image is acquired successively a preset particular number oftimes. Furthermore, in the camera system according to the thirdembodiment, the process described above is performed repeatedly.

Thus, in the camera system according to the third embodiment, areference image is acquired periodically.

Let it be assumed by way of example that a problem occurs in a mechanismfor determining whether to acquire a reference image or a multipleexposure image, or an error occurs in the determination. In such a case,there is a possibility that a multiple exposure image is acquired in asituation in which a reference image should be acquired. If such anerror occurs, a reference image is not acquired for a certain period,which may make it difficult to properly extract an object from themultiple exposure image.

In contrast, in the camera system according to the third embodiment, itis possible to prevent an occurrence of a situation in which a referenceimage is not acquired over a certain period.

Fourth Embodiment 4.1 Overview

A camera system according to a fourth embodiment is described below.

In the first embodiment described above, a feature value of a subject isextracted from a reference image acquired by a normal exposure.

In contrast, in the fourth embodiment, a feature value is extracted froman image obtained by first N exposures in a multiple exposure. N is aninteger set to a value greater than 1 and smaller than the number ofexposures performed in acquiring a multiple exposure image.

4.2 Configuration

The camera system according to the fourth embodiment may be similar inconfiguration to the camera system 10 according to the first embodiment,and thus a description of the constituent elements is omitted.

4.3 Operation

In the camera system according to the fourth embodiment, a fourth movingobject detection process is performed instead of the first moving objectdetection process according to the first embodiment.

The fourth moving object detection process is described below withreference to drawings.

FIG. 17 is a flow chart illustrating a fourth moving object detectionprocess.

As shown in FIG. 17, the fourth moving object detection process isdifferent from the first moving object detection process according tothe first embodiment in that the process from step S100 to step S120 isremoved, a process from step S400 to step S460 is added, the process instep S180 is replaced by a process in step S480, and the process in stepS190 is replaced by a process in step S490.

Thus, the following explanation focuses on the process from step S400 tostep S460, the process in step S480, the process in step S490, andrelated processes.

As shown in FIG. 17, when the fourth moving object detection processstarts, the imaging device 20 starts capturing a multiple exposure imageby performing a multiple exposure (step S400). At a point of time whenan N-th exposure in a multiple exposure is ended, a pixel value isnondestructively read out from each pixel 101 (steps S410, S420). Thus,an image is acquired by as many N exposures. Hereinafter, this image isreferred to as a “nondestructive image”. Although the value of N is notlimited to 1, it is desirable that N=1.

In the following description, a “nondestructive reading mode” denotes anoperation mode in which a nondestructive image is acquired at a point oftime when an N-th exposure in a multiple exposure is ended, while a“normal reading mode” denotes an operation mode in which anondestructive image is not acquired in the middle of a multipleexposure. When the operation modes are defined in the above-describedmanner, the process in step S400 is a process in which capturing amultiple exposure image in the nondestructive reading mode is started.

FIG. 18 illustrates an example of operation timing of a multipleexposure in two frame periods. Note that in this example, it is assumedthat each pixel 101 of the imaging device 20 includes a photoelectricconversion element using a photoelectric conversion film.

In FIG. 18, a graph of a nondestructive reading signal represents achange in a pulse signal that controls the timing of nondestructivereading.

The nondestructive reading signal rises up at a point of time when anN-th exposure is ended in a frame period in which the nondestructivereading mode is employed as the operation mode. In the specific exampleshown in FIG. 18, the multiple exposure is performed 3 times and N=1.

The imaging device 20 acquires a nondestructive image at a point of timewhen the nondestructive reading signal rises up. Thus, the imagingdevice 20 acquires a nondestructive image and a multiple exposure imagein the first frame period. Furthermore, in the second frame period, theimaging device 20 acquires only a multiple exposure image withoutacquiring a nondestructive image.

Referring again to FIG. 17, the fourth moving object detection processis further described below.

When the nondestructive image is acquired, the image processor 30extracts a feature value of each of all subjects from the acquirednondestructive image (step S430). Based on each extracted feature value,the recognition is performed to identify what each subject is (stepS440). When the acquisition of the multiple exposure image is ended(step S450), the processing flow proceeds to step S130 to perform theprocess in step S130 and processes in following steps.

In a case where it is determined in the process in step S170 that themoving object detection mode is not to be ended (No in step S170), thecontrol circuit 40 determines whether to acquire a multiple exposureimage in the nondestructive reading mode or in the normal reading modein a next frame period (step S480).

The determination as to whether to acquire a multiple exposure image inthe nondestructive reading mode or in the normal reading mode in a nextframe period may be made in a similar manner to that employed in stepS180 in the first moving object detection process according to the firstembodiment. More specifically, in the explanation on the process in stepS180, “a normal exposure is performed” is read as “a multiple exposureimage is acquired in the nondestructive reading mode, and “a multipleexposure is performed” is read as “a multiple exposure image is acquiredin the normal reading mode”. A further detailed description thereof isomitted.

In a case where it is determined in the process in step S480 that amultiple exposure image is to be acquired in the nondestructive readingmode in the next frame period (Yes in step S490), the processing flowreturns to step S400 to repeat the process from step S400.

In a case where in the process in step S480, it is not determined that amultiple exposure image is to be acquired in the nondestructive readingmode in the next frame period, that is, in a case where it is determinedthat a multiple exposure image is acquired in the normal reading mode inthe next frame period (No in step S490), acquiring the multiple exposureimage in the normal reading mode is started (step S460). When theacquisition of the multiple exposure image is ended (step S450), theprocessing flow proceeds to step S130 to perform the process in stepS130 and processes in following steps.

4.4 Supplemental Remarks

The camera system according to the fourth embodiment extracts a featurevalue from a nondestructive image acquired in the nondestructive readingmode.

Thus, the camera system according to the fourth embodiment, is capableof accurately identifying a location of a subject in a multiple exposureimage without providing an additional frame period in which a normalexposure is performed. Furthermore, it is not necessary to provide aframe period in which a reference image is acquired, and thus it isallowed to successively acquire multiple exposure images and thus it ispossible to continuously detect a moving object.

Fifth Embodiment 5.1 Overview

A camera system according to a fifth embodiment is described below.

In the first embodiment described above, the sensitivity is maintainedconstant over all different exposure periods when a multiple exposureimage is acquired.

In contrast, in the fifth embodiment, the sensitivity is varieddepending on exposure periods in a multiple exposure operation.

5.2 Configuration

The camera system according to the fifth embodiment may be similar inconfiguration to the camera system 10 according to the first embodiment,and thus a description of the constituent elements is omitted.

5.3 Operation

In the camera system according to the fifth embodiment, a five movingobject detection process is performed instead of the first moving objectdetection process according to the first embodiment.

The five moving object detection process is described below withreference to drawings.

The five moving object detection process is different from the firstmoving object detection process according to the first embodiment inthat a sensitivity of each exposure in a multiple exposure for acquiringa multiple exposure image is set to either one of two differentsensitivities. The flow chart illustrating the five moving objectdetection process is the same as the flow chart of the first movingobject detection process according to the first embodiment, and thus adescription of each step is omitted.

FIGS. 19A to 19C each illustrate an example of operation timing in anormal exposure and a multiple exposure in two frame periods. Note thatit is assumed here that each pixel 101 of the imaging device 20 includesa photoelectric conversion element using a photoelectric conversionfilm. In FIGS. 19A to 19C, a normal exposure is performed in a firstframe period while a multiple exposure is performed in a second frameperiod.

As shown in FIGS. 19A to 19C, during the second frame period in which amultiple exposure is performed, the voltage applied to the transparentelectrode 110A is set to either V1 or V2 except when the voltage is atthe low level, where V1 is equal to the high level and V2 is lower thanV1 and higher than the low level.

As described above, when the bias voltage applied to the photoelectricconversion layer 110C is relatively high, the photoelectric conversionefficiency is relatively high, while when the bias voltage applied tothe photoelectric conversion layer 110C is relatively low, thephotoelectric conversion efficiency is relatively low. Therefore, in aperiod in which the voltage applied to the transparent electrode 110A isV1, the sensitivity is higher than in a period in which the voltageapplied to the transparent electrode 110A is V2.

5.4 Supplementary Remarks

The camera system according to the fifth embodiment is capable ofchanging the luminance of images such that the luminance of an image ofa subject acquired by a specific exposure in a multiple exposure isdifferent from luminance of images of the same subject.

For example, as shown in FIG. 19A, the voltage applied to thetransparent electrode 110A is set to V1 only during a last exposureperiod in a multiple exposure, while the voltage applied to thetransparent electrode 110A is set to V2 during the other exposureperiods. This makes it possible to obtain higher luminance for an imageof a subject acquired by a last exposure than for the other images ofthe same subject in a multiple exposure image. Thus, it becomes possibleto relatively easily identify the image of the subject acquired by thelast exposure in the multiple exposure image.

For example, as shown in FIG. 19B, the voltage applied to thetransparent electrode 110A is set to V1 only during a first exposureperiod in a multiple exposure, while the voltage applied to thetransparent electrode 110A is set to V2 during the other exposureperiods. This makes it possible to obtain higher luminance for an imageof a subject acquired by a first exposure than for the other images ofthe same subject in a multiple exposure image. Thus, it becomes possibleto relatively easily identify the image of the subject acquired by thefirst exposure in the multiple exposure image.

For example, as shown in FIG. 19C, the voltage applied to thetransparent electrode 110A is set to V1 only during a first exposureperiod and a last exposure period in a multiple exposure, while thevoltage applied to the transparent electrode 110A is set to V2 duringthe other exposure periods. This makes it possible to obtain higherluminance for images of a subject acquired by a first exposure and alast exposure than for the other images of the same subject in amultiple exposure image. Thus, it becomes possible to relatively easilyidentify the images of the subject acquired by the first exposure andthe last exposure in the multiple exposure image.

In any example described above, the camera system according to the fifthembodiment is capable of more easily calculating a direction of movementof the subject than is possible by the camera system according to thefirst embodiment.

Sixth Embodiment 6.1 Overview

A camera system according to a sixth embodiment is described below.

In the fifth embodiment described above, when a multiple exposure imageis acquired, the sensitivity in each exposure is allowed to be set toone of two values.

In contrast, in the sixth embodiment, when a multiple exposure image isacquired, the sensitivity in each exposure is allowed to be set to oneof three or more values.

6.2 Configuration

The camera system according to the sixth embodiment may be similar inconfiguration to the camera system 10 according to the first embodiment,and thus a description of the constituent elements is omitted.

6.3 Operation

In the camera system according to the sixth embodiment, a sixth movingobject detection process is performed instead of the five moving objectdetection process according to the fifth embodiment.

The sixth moving object detection process is described below withreference to drawings.

The sixth moving object detection process is different from the firstmoving object detection process according to the first embodiment inthat when a multiple exposure image is acquired, the sensitivity in eachexposure is set to one of three different values. The flow chartillustrating the sixth moving object detection process is the same asthe flow chart of the first moving object detection process according tothe first embodiment, and thus a description of each step is omitted.

FIGS. 20A and 20B each illustrate an example of operation timing of anormal exposure and a multiple exposure in two frame periods. Note thatit is assumed here that each pixel 101 of the imaging device 20 includesa photoelectric conversion element using a photoelectric conversionfilm. In FIGS. 20A and 20B, a normal exposure is performed in a firstframe period and a multiple exposure is performed in a second frameperiod.

As shown in FIGS. 20A and 20B, during the second frame period in which amultiple exposure is performed, the voltage applied to the transparentelectrode 110A is set to one of three voltages V1, V2, and V3 exceptwhen the voltage is at the low level, where V1 is equal to the highlevel, V2 is lower than V1 and higher than V2, and V3 is lower than V2and higher than the low voltage. In a period in which the voltageapplied to the transparent electrode 110A is V1, the sensitivity ishigher than in a period in which the voltage applied to the transparentelectrode 110A is V2, while in a period in which the voltage applied tothe transparent electrode 110A is V2, the sensitivity is higher than ina period in which the voltage applied to the transparent electrode 110Ais V3.

In the following explanation of the sixth embodiment, it is assumed byway of example that the voltage applied to the transparent electrode110A is set to one of three values V1, V2, and V3, although the voltageapplied to the transparent electrode 110A may be set to one of four ormore values.

6.4 Supplementary Remarks

The camera system according to the sixth embodiment is capable ofchanging the luminance among a plurality of images acquired for the samesubject in a multiple exposure image.

For example, as shown in FIG. 20A, when a multiple exposure image isacquired, the voltage applied to the transparent electrode 110A in eachexposure may be varied such that the later the exposure period thehigher the applied voltage. This makes it possible to obtain luminancefor the images of the same subject acquired in the multiple exposureimage such that the later the image the higher the luminance. Thus, itbecomes possible to relatively easily identify the time sequential orderof the images of the same subject acquired in the multiple exposureimage. Thus, it becomes possible to more easily calculate the directionof movement of the subject than in the case where the sensitivity is notchanged among exposures when a multiple exposure image is acquired.

For example, as shown in FIG. 20B, when a multiple exposure image isacquired, the voltage applied to the transparent electrode 110A in eachexposure may be varied such that the later the exposure the lower thevoltage. This makes it possible to obtain luminance for the images ofthe same subject acquired in the multiple exposure image such that thelater the image the lower the luminance. Thus, it becomes possible torelatively easily identify the time sequential order of the images ofthe same subject acquired in the multiple exposure image. Thus, itbecomes possible to more easily calculate the direction of movement ofthe subject than in the case where the sensitivity is not changed amongexposures when a multiple exposure image is acquired.

Seventh Embodiment 7.1 Overview

A camera system according to a seventh embodiment is described below.

In the first embodiment described above, when a multiple exposure imageis acquired, an exposure period and a non-exposure period are providedalternately and an exposure is performed a plurality of times.

For example, in a case where each pixel 101 of the imaging device 20 isconfigured to include a photoelectric conversion element using aphotoelectric conversion film, each non-exposure period is a period inwhich the voltage VITO applied to the transparent electrode 110A is atthe low level (see FIG. 6). On the other hand, in a case where eachpixel 101 of the imaging device 20 is configured to include a photodiodefunctioning as a photoelectric conversion element, each non-exposureperiod is a period from a certain point of time when Vtrs falls down toa later point of time when Vdr falls down next time (see FIG. 8).

In contrast, in the seventh embodiment, when a multiple exposure imageis acquired, a high-sensitivity exposure and a low-sensitivity exposureare performed alternately.

7.2 Configuration

The camera system according to the seventh embodiment may be similar inconfiguration to the camera system 10 according to the first embodiment,and thus a description of the constituent elements is omitted.

7.3 Operation

In the camera system according to the seventh embodiment, a seventhmoving object detection process is performed instead of the first movingobject detection process according to the first embodiment.

The seventh moving object detection process is described below withreference to drawings.

The seventh moving object detection process is different from the firstmoving object detection process according to the first embodiment inthat when a multiple exposure image is acquired, the high-sensitivityexposure and the low-sensitivity exposure are performed successively andalternately. The flow chart illustrating the seventh moving objectdetection process is the same as the flow chart of the first movingobject detection process according to the first embodiment, and thus adescription of each step is omitted.

FIG. 21 illustrates an example of operation timing in a normal exposureand a multiple exposure in two frame periods. Note that in this example,it is assumed that each pixel 101 of the imaging device 20 is configuredto include a photoelectric conversion element using a photoelectricconversion film. In FIG. 21, a normal exposure is performed in a firstframe period and a multiple exposure is performed in a second frameperiod.

As shown in FIG. 21, during the second frame period in which a multipleexposure is performed, the voltage applied to the transparent electrode110A is set to V1 and V4 alternately and repeatedly where the imagingdevice 20 has a particular sensitivity when the voltage is set to V1,while V4 is lower than V1 and higher than the low level.

7.4 Supplementary Remarks

In the camera system according to the seventh embodiment, when amultiple exposure image is acquired in the seventh moving objectdetection process, the high-sensitivity exposure period, in which thevoltage applied to the transparent electrode 110A is set to V1, and thelow-sensitivity exposure period, in which the voltage applied to thetransparent electrode 110A is set to V2, are repeatedly continuously andalternately. As a result, a trajectory of a movement of a subject isformed on the acquired multiple exposure image.

FIG. 22A illustrates an example of a multiple exposure image acquiredvia the first moving object detection process according to the firstembodiment. FIG. 22B illustrates an example of a multiple exposure imageacquired via the seventh moving object detection process according tothe seventh embodiment.

In FIG. 22A, images of the same subject are formed at separatelocations. In contrast, in FIG. 22B, images are acquired bylow-sensitivity exposures in periods between adjacent high-sensitivityexposures, and these images form a trajectory of the movement of thesubject.

As described above, a trajectory of a movement of a subject appears on amultiple exposure image acquired via the seventh moving object detectionprocess. To search for an image of a subject, it is necessary to searchan area where the subject is predicted to move in a particulardirection. In the present embodiment, by tracking the trajectory of thesubject, it is possible to narrow a searching area for each of aplurality of images of the same subject acquired by high-sensitivityexposures. Thus, the camera system according to the seventh embodimentis capable of relatively easily detecting each of a plurality of imagesof the same subject acquired by high-sensitivity exposures in a multipleexposure image.

Supplements

The present disclosure has been described above by way of example withreference to the first to seventh embodiments. Note that the presentdisclosure is not limited to those embodiments described above, but manyvarious modifications, changes, replacements, additions, removals, etc.,may be applicable to the embodiments without departing from the spiritand scope of the present disclosure.

Some examples of modifications of the embodiments of the presentdisclosure are described below.

1. In the first embodiment, the traveling control system 1 includes theconstituent elements shown in FIG. 1. However, in the presentdisclosure, the configuration of the traveling control system is notlimited to the example shown in FIG. 1 as long as functions similar tothose provided by the traveling control system 1 are achieved.

FIG. 23 is a block diagram illustrating a configuration of a travelingcontrol system 1 a according to a modified embodiment.

As shown in FIG. 23, the traveling control system 1 a is different fromthe traveling control system 1 in that the motion detection unit 32 isremoved and the ISP 31 is replaced by an ISP 31 a. As a result of thedifferences described above, the image processor 30 is replaced by animage processor 30 a, and the camera system 10 is replaced by a camerasystem 10 a.

The ISP 31 a provides the functions of the ISP 31 and the functions ofthe motion detection unit 32. Thus, the traveling control system 1 a canprovide functions similar to those provided by the traveling controlsystem 1.

2. In the third embodiment, the determination as to whether successiveacquisition of multiple exposure images is to be continued or thesuccessive acquisition of multiple exposure images is to be ended and areference image is to be acquired is made based on the number ofsuccessive acquisitions of multiple exposure images. In the camerasystem according to the third embodiment, the determination describedabove may be made based on another criterion.

For example, the determination described above may be made based on atime during which acquiring of a multiple exposure image is continued.For another example, the determination described above may be made basedon a traveling distance of a vehicle on which the imaging device 20 isinstalled.

The camera system and the traveling control system according to thepresent disclosure can be used in a wide variety of systems in which animage is acquired.

What is claimed is:
 1. A camera system comprising: an imaging devicethat captures a first image by a normal exposure including only oneexposure and that captures a second image by a multiple exposureincluding a plurality of exposures; and a control circuit, wherein theimaging device captures the second image in a first frame period, andthe control circuit determines, based on the second image captured inthe first frame period, whether to capture an image by the normalexposure or capture an image by the multiple exposure in a second frameperiod following the first frame period.
 2. The camera system accordingto claim 1, further comprising an image processor, wherein in a casewhere the image processor cannot detect a first object in the secondimage captured in the first frame period, the control circuit determinesto capture an image by the normal exposure in the second frame period.3. The camera system according to claim 1, further comprising an imageprocessor, wherein in a case where the image processor detects a firstobject in the second image captured in the first frame period, thecontrol circuit determines to capture an image by the multiple exposurein the second frame period.
 4. The camera system according to claim 1,wherein the multiple exposure includes a first exposure and a secondexposure different from the first exposure, and a sensitivity in thefirst exposure is different from a sensitivity in the second exposure.5. The camera system according to claim 1, wherein a sensitivity in aperiod between adjacent two exposures out of the multiple exposure isgreater than zero and less than a sensitivity in each of the adjacenttwo exposures.
 6. The camera system according to claim 1, furthercomprising an image processor that calculates, based on the secondimage, at least a direction of movement, a velocity, or an accelerationof a first object captured by the imaging device.
 7. A camera systemcomprising: an imaging device that captures a first image by a normalexposure including only one exposure and that captures a second image bya multiple exposure including a plurality of exposures; and a controlcircuit, wherein the imaging device captures the first image in a firstframe period, and the control circuit determines, based on the firstimage captured in the first frame period, whether to capture an image bythe normal exposure or capture an image by the multiple exposure in asecond frame period following the first frame period.
 8. The camerasystem according to claim 7, further comprising an image processor,wherein in a case where the image processor cannot detect a first objectin the first image captured in the first frame period, the controlcircuit determines to capture an image by the normal exposure in thesecond frame period.
 9. The camera system according to claim 7, furthercomprising an image processor, wherein in a case where the imageprocessor detect a first object in the first image captured in the firstframe period, the control circuit determines to capture an image by themultiple exposure in the second frame period.
 10. The camera systemaccording to claim 7, wherein the multiple exposure includes a firstexposure and a second exposure different from the first exposure, and asensitivity in the first exposure is different from a sensitivity in thesecond exposure.
 11. The camera system according to claim 7, wherein asensitivity in a period between adjacent two exposures out of themultiple exposure is greater than zero and less than a sensitivity ineach of the adjacent two exposures.
 12. The camera system according toclaim 7, further comprising an image processor that calculates, based onthe second image, at least a direction of movement, a velocity, or anacceleration of a first object captured by the imaging device.
 13. Acamera system comprising: an imaging device that captures a first imageby a normal exposure including only one exposure and that captures asecond image by a multiple exposure including a plurality of exposures;and a control circuit, wherein the control circuit causes the imagingdevice to capture the first image in a frame period, to capture thesecond image in each of one or more frame periods following the frameperiod, and to capture the first image in a frame period following theone or more frame periods.
 14. The camera system according to claim 13,wherein the multiple exposure includes a first exposure and a secondexposure different from the first exposure, and a sensitivity in thefirst exposure is different from a sensitivity in the second exposure.15. The camera system according to claim 13, wherein a sensitivity in aperiod between adjacent two exposures out of the multiple exposure isgreater than zero and less than a sensitivity in each of the adjacenttwo exposures.
 16. The camera system according to claim 13, furthercomprising an image processor that calculates, based on the secondimage, at least a direction of movement, a velocity, or an accelerationof a first object captured by the imaging device.